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Abstract — Consider the following information theoretic 
setup wherein independent codebooks of N correlated ran- 
dom variables are generated according to their respective 
marginals. The problem of determining the conditions on 
the rates of codebooks to ensure the existence of at least 
one codeword tuple which is jointly typical with respect 
to a given joint density (called the multivariate covering 
lemma) has been studied fairly well and the associated rate 
regions have found applications in several source coding 
scenarios. However, several multiterminal source coding 
applications, such as the general multi-user Gray-Wyner 
network, require joint typicality only within subsets of 
codewords transmitted. Motivated by such applications, 
we ask ourselves the conditions on the rates to ensure 
the existence of at least one codeword tuple which is 
jointly typical within subsets according to given per subset 
joint densities. This report focuses primarily on deriving a 
new achievable rate region for this problem which strictly 
improves upon the direct extension of the multivariate 
covering lemma, which has quite popularly been used in 
several earlier work. Towards proving this result, we derive 
two important results called 'subset typicality lemmas' 
which can potentially have broader applicability in more 
general scenarios beyond what is considered in this report. 
We finally apply the results therein to derive a new 
achievable region for the general multi-user Gray-Wyner 
network. 

Index Terms — Typicality within subsets, Multivariate 
covering lemma, Multi-user Gray-Wyner network 

I. Introduction 
Consider a scenario where independent codebooks 
of N random variables (Xx, X2, ■ ■ ■ , Xn) are gener- 
ated according to some given marginal distributions at 
rates (Rx, R2, . . . ,Rn) respectively. Let Sx, <?2 • • ■ Sm 
be M subsets of {1,2,..., N} and let the joint distribu- 
tions of (Xx, X2, . . . , Xn) within each subset, consistent 
with each other and with the marginal distributions, be 
given. We ask ourselves the conditions on the rates 
(Rx,R2, ■ ■ ■ ,Rn) (achievable region) so that the prob- 
ability of finding one codeword from each codebook, 
such that the codewords are all jointly typical within 



subsets Sx,S2 ■ ■ ■ Sm according to the given per subset 
joint distributions, approaches 1. We denote the given 
probability distribution over subset Si by P({X}s f )- The 
conditions on the rates when <Sj = {1, . . . , N}, i.e, when 
the joint distribution over all the random variables is 
given, can be derived using standard typicality arguments 
and is quite popularly called as the multivariate cover- 
ing lemma HJ, [20. It says that for any joint density 
over (Xx, X2, ■ ■ ■ , Xn), if the codebooks are generated 
according to the respective marginals, the probability of 
not finding a jointly typical codeword tuple approach 
if VJC{l,2,...,iV}: 
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where {X} j denotes the set Xi : i € J and H(P) 
denotes the entropy of any distribution P. 

A fairly direct extension of the multivariate covering 
lemma, to the more general scenario of arbitrary subsets 
5i,52 . . - Sm, which has been quite popularly used in 
several information theoretic scenarios, such as 10, [@], 
0, @, Q, can be described as follows. Fix any joint 
density P(Xx, X2, . . . , Xn) such that: 



P({X} Sj ) = P Sj ({X} Sj ) Vj 



(2) 



i.e, it satisfies the given joint distributions within subsets 
Sj Vj. Then the set of all rate tuples satisfying the fol- 
lowing conditions are achievable, \/J C {1, 2, . . . , N}: 
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H{P({X}j)) 



(3) 



The convex closure of all achievable rate tuples, over 
all such joint densities P satisfying the given per subset 
densities is an achievable region for the problem. We 
denote this region by lZ a . Our primary objective in 

'We note that the underlying principles and proofs of multivariate 
covering lemma appeared much earlier in the literature, for example 
t3l . However the nomenclature and the general applicability of the 
underlying ideas have been elucidated quite clearly in UJ 



this report is to show that the rate region in d3) with 
the individual functionals set to their respective maxima 
subject only to their specific exact constraints is, infact, 
achievable. Specifically we show that, each of the terms 
H(P({X}j)) can be replaced with the corresponding 
maximum entropy functionals H*(P({X}j)) subject to 
only the constraints pertinent to subsets of {X}j. This 
allows us to achieve simultaneous optimum of all the 
functionals leading to a strictly larger achievable region 
than R a . Towards proving this result, we establish two 
important lemmas, namely 'subset typicality lemmas', 
which may prove to have much wider applicability in 
general scenarios beyond the scope of this report. 

Scenarios depicted in the above example, where typi- 
cality within subsets of codewords is sufficient for decod- 
ing, arise quite frequently in several multi terminal source 
coding setups. One of the most typical examples is the 
multi-user generalization of the Gray-Wyner network ||8] 
discussed in section [III] where the encoder observes K 
random variables and there are K sinks, each decoding 
one of the random variables upto a prescribed distortion 
constraint The most general setting involves 2 K — 1 
branches (encoding rates), each being sent to a unique 
subset of the decoders. Observe that it is sufficient if all 
the codewords being sent to sink i are jointly typical with 
the i'th source sequence and enforcing joint typicality of 
all the codewords in an unnecessary restriction. Similar 
settings arise in the context of dispersive information 
routing of correlated sources d, fusion coding and 
selective retrieval in a database [6| and in several other 
scenarios which can be considered as particular cross- 
sections of the general L— channel 'multiple descrip- 
tions' (MD) problem Q, 0. We note that, in this report, 
we demonstrate the workings of the underlying principle 
in the context of the example we described above. 
However it is important to note that the results we derive 
have implications in a wide variety of problems involving 
optimization of multiple functionals, each depending on 
a subset of the random variables, subject to constraints 
on their joint distributions. 

II. Main Results 

In this section, we first establish the subset typicality 
lemmas which will finally lead to Theorems Q] and |2] 

2 We note that [ 9 1 considers a particular generalization of the Gray- 
Wyner network to multiple users with applications in information 
theoretic security where a unique common branch is sent to all the 
decoders along with their respective individual rates. However we 
assert that the most general extension of the 2 user Gray-Wyner 
network will involve a combinatorial number of branches, each being 
sent to a unique subset of the decoders. 



showing strictly larger achievable rates compared to 1Z a . 
Throughout the report, we use the following notation. 
n independent and identically distributed (iid) copies 
of a random variable and its realizations are denoted 
by Xq and Xq respectively. Length n, e-typical set 
of any random variable X, with distribution P(X) is 
denoted^by T?(P{X)). Throughout the report, for any 
set S, we use the shorthand {U}s to denote the set 
{Ui : i € <S}. Note the difference between t/123, which 
is a single random variable and {C/}i23, which is the 
set of random variables {U\, U2, U3}. In the following 
Lemmas, we use the notation P(A) = 2~ nR to denote 

2 „ n{R+S ( e )) < p ^ < 2 - n{R - S (e)) for some 6 ^ _^ 

as e — > 0. To avoid resolvable but unnecessary com- 
plications, we further assume that there exists at least 
one joint distribution consistent with the prescribed per 
subset distributions for S\, S2, ■ ■ ■ , <Sm- 

A. Subset Typicality Lemmas 

Lemma 1. Subset Typicality Lemma :Let 
(Xi, X2, ■ ■ ■ -Xjv) b e N random variables taking 
values on arbitrary finite alphabets (X\, X2, . . . Xn) 
respectively. Let their marginal distributions be 
Pi (Xi ) , P 2 {X 2 ) . . . , Pn (Xn ) respectively. Let 
S\,S2---Sm be M subsets of {1,2, ...,iV} and 
for all j € {1, 2, . . . , M}, let Pg. ({X}s ) be any given 
joint distribution for {X}$. consistent with each other 
and with the given marginal distributions. Generate 
sequences X™, xV^ ■ ■ ■ x^, each independent of the other, 
where x™ is drawn iid according to the marginal 
distribution Pj(Xj), i.e., xf ~ IIILi Pi( x il)- Then, 

P e V 1 {PsM^W) ,Vj e {1... M}) 

i 2-»<E£i H{Xi)-H{P»)) (4) 

where P* is a distribution over (X\, X2 ■ ■ ■ , Xn) which 
satisfies: 

P* = argmaxF (p) (5) 

subject to P({X} Sj ) = P Sj ({X} Sj ) Vj G {1 . . . M}. 

This Lemma essentially says that the total number of 
sequence tuples (x™, x^ . . . x N ) generated according to 
their respective marginals which are jointly e— typical ac- 
cording to Psi ({X}s. ) Mi within subsets <Si, S2 ■ ■ ■ Sm, 
is approximately 2 nH< - p *^ where P* is the maximum 
entropy distribution subject to the constraint that the 
joint density within subset Si is Ps j ({X}s. )Vj. 

3 The parenthesis is dropped whenever it is obvious 



Proof: To prove this Lemma, we resort to Sanov's 
theorem ([10] Theorem 11.4.1) from the theory of large 
deviations. Sanov's theorem states that for any distribu- 
tion Q(X) and for any subset of probability distributions 
£ C V, where V denotes the universe of the PMFs over 
the alphabets of X: 

Q n (£) = 2- nD ( p *HQ) (6) 

for sufficiently large n, where P* is the distribution 
closest in relative entropy to Q in £ and Q n (£) denotes 
the probability that an iid sequence generated according 
to Q(X) is e— typical with respect to some distribution 
in £. We set Q(-) = UiLi p i( X i) and £ as the set 
of all distributions over (X\, X2, . . . Xn) satisfying the 
given constraints. Then it follows from Sanov's theorem 
that the probability of (x™ ...x 1 ^) being e— typical ac- 
cording to some distribution satisfying the given con- 
straints is approximately 2~ n - D ( p *l n<=i Pi(Xi))^ w h ere 
P* is the distribution having minimum relative entropy 
to Y[iLiPi(Xi) and satisfying the given constraints. 
However, all such distributions have the same marginal 
distributions Pi(X±), Pi{X2) ■ ■ ■ , Pi(Xn)- Hence mini- 
mizing relative entropy is equivalent to maximizing the 
joint entropy leading to P* as defined in ([5]>. Therefore 
we have: 

P (MS, e 77(WsJ, Vi) = Q n (£) = 2- nD ^ p '^ 

(7) 

where the last equality follows because P* satisfies the 
given marginals. ■ 
We note that a particular instance of Lemma [T] was 
derived in ifTTI . However, as it turns out, for the setup 
they consider, this Lemma does not help in deriving an 
improved achievable region. In the following lemma, we 
establish the conditional version of Lemma Q] Note that 
Lemma [2] is not used in proving Theorems [T] or |2] but 
will play a crucial role in the application of these results 
to more general multi-terminal source coding scenarios 
(as we will see in section [TITb. 

Lemma 2. Conditional Subset Typicality Lemma :Let 
random variables (X\, X2, ■ ■ ■ Xjy), sets S\, S2 ■ ■ ■ Sm 
and joint densities P$ ({X}$ ) be defined as in Lemma 
UJ Let the sequences (x™ . . . x" 1 ^) be generated such that 
each sequence is generated conditioned on a subset of 
already generated sequences {x}^. and independent of 
the rest, where (i, Aj) € Sj for some j € {!,••• , M}. 



Then we have: 

P({x}lel?({x}s i )Vie{i...M})± 

where P* satisfies (J5]). 

Proof: The proof follows in very similar lines to 
that of Lemma[T]by setting Q(-) = n^Ii P{Xi\X Az ), as 
conditioning on only introduces further constraints, 
which are redundant, as Ps ] {{X}s j ) are consistent with 
each other and (i,Ai) £ Sj for some j 6 {1, . . . ,M}. 

m 

B. Simultaneous Optimality of Functionals 

In this section we will show that simultaneous opti- 
mality of all function H(P({X}j)) is in fact achievable 
leading to a new achievable rate region for the problem 
stated in the introduction. 

Theorem 1. Let random variables (Xi, X2, ■ ■ ■ X^), 
sets S\, S2 ■ ■ ■ Sm and joint densities P$ ({X}s ) be 
defined as in Lemma [7J For each i 6 {1,2..., M}, let 
x"(mj) m j € {1, . . . , 2 nRi } be independent sequences 
drawn iid according to the respective marginals, i.e., 
Xiirrii) ~ nr=i Pi{xu{mi)) Vm, G {1, ... , 2 nR >}. Then 
Ve > 0, 3<5(e) such that 5(e) — > as e — > and, 

p([x}% (W5,) £ T? {P Sj [{X} Sj )) Vj 

for some {mi, 1712 ■ ■ ■ ,mjv}^ > 1 — 5(e) (9) 

if, i?2 ■ ■ ■ , Rn) satisfy the following conditions 
VJC{l,2,...,iV}: 

iej iej 

where, 

H*({X}j)= max H (P({X}jj) (11) 

where P({X}j) satisfies: 

P {{X}jn S] ) = P {{Xjjns^j £ {1 . . . Af} (12) 

We denote the rate region in diOD by 1Z*. 

Remark 1. Note that H*({X}j) = H(P({X}j)) if 
J C Sj for some Sj. Hence for all J such that J C Sj 
for some j, the corresponding inequalities in Theorem 
Q] and equations © are the same. However this theorem 
asserts that for every other J , the functionals in ((2]) can 
be replaced with the 'maximum joint entropy' subject 



to the given subset distributions which involve only the 
random variables {X}j. It is very important to note 
that the maximum entropy distributions for two different 
subsets Jjj and Xj 2 , J\,J?. C {1, 2, . . . , N}, may 
not even correspond to any valid joint distribution over 
(Xi,X2, ■ ■ ■ ,Xn). This is precisely what provides the 
additional leeway in achieving points which are strictly 
outside (O as illustrated in Theorem [2] A pictorial 
representation of the above theorem is shown in Fig. 

m 

Proof: We are interested in finding conditions 
on rates so that the probability in © approaches 
1. Denote the event £ = p{mi, . . . , tun} : 
{*}% {{m}s 3 ) 6 7? (P Sj ({X} Sj )) Vj. We want to 
make P{£) ->• 0. Let M denote the set {1,2,..., N} 
and let (mi, m2, • • • , m^) = {m}Af be an index 
tuple, one from each codebook, such that m 8 G 
{1, . . . ,2 nRi }. Let £({m}x) denote the event that 
Ms, (Kk) e V 1 (Ps,({X} s .)) Vj. Define random 
variables x({ m }N) sucn that: 



x{{™]n) 



1 if £({m}_\f) occurs 
else 



(13) 



and random variable \ = J2{m}_v x({ m }jv)- Then we 
have P{£) = P(x = 0). From Chebyshev's inequality, 
it follows that: 

P{£) = P( X = 0)<P [\x - E( X )\ > E( X )/2] (14) 

4Var( X ) 4 ( E U 2 ) ~ ( E U)f 



< 



(E(x)Y 



We next bound E(x) and E(x 2 ) using Lemma Q] First 
we write E(x) as: 



E( X ) 



2 n ^ R >P(£({mU)) (15) 



for any {m}j^ because all the sequences are drawn in- 
dependent of each other. Next towards bounding E(x 2 ), 
note that: 

E (X 2 ) = E E P (£({mU),£({lW)) (16) 

{m} N {l}^ 

Let {m}g = {1}q and {m}^-Q / 0}jv"-Q for some 
Q Q J\f,Q ^ <fi where denotes a null-set. Then, 

P{£({m} M ),£{{l} M )) = jp(£({m} e )) 

P(^({m}^)|^({m} s )) 2 l (17) 



where £({ m }Q) denotes the event that 

toS,n C (W^ns) e TT (^n Q ({X} 5j n S )) 
Vj, as conditional on {^}g({m}Q), sequences 

M/V-qCW-A^-s) and W/V-Q (WaA-q) are drawn 
independently from the same distribution. The above 
expression can be rewritten as: 



P (£ ({mU), £({IU)) = P (S({m} Q )) 



P 



{£{{m} N )) \ 2 \ 
P(S({m} Q ))J j 

If Q = <t>, we have P (£{{m} N ),£({l} N )) 
(P {£{{m}^f))) 2 . Hence, we can write Var(x) as: 



(18) 



Var(x) = E 1 2 



xP(£({m} Q )) 



P(£({m} Q )). 



(19) 



Note that the Q = <fi term gets cancelled with the 
\E{x)) 2, terms in Var(x) (see O for a similar ar- 
gument). 

On substituting (fT5T ) and ( fT9l in (fT4l ). and noting that 

for any Q C TV, Q ^ 0, we can write P(£"({m}^)) = 



P(£({m} Q )) 



P(g({mV)) 



, we have: 



P(e({m} Q )) 

P(£)<A Y, 2^«*(P(£({m} a ))) 



-i 



(20) 



Next, invoking Lemma [T] we bound P(£({m}g)) as: 
P(£({m} Q )) > 2 - n (^aH(X,)-H'({X} a ))ynS(e) 

(21) 

On substituting (EB in (|20]>, it follows that P{£) 
as n — > oo if Pj satisfy (fTOl . ■ 

C. SYr/cf Improvement 

Theorem 2. (7) 77ie region in Theorem\J} subsumes the 
region in 0. i.e, 

Tl a C 7^ (22) 

( //) There exist scenarios for which the region in Theorem 
\J\can be strictly larger than the region in $3}. i.e., 



(23) 



Proof: The first half of the Theorem follows directly 
because H*({X}j) > H({X}j) MJ for any joint dis- 
tribution satisfying the given distributions within subsets. 
To prove (ii) we provide an example for which lZ* a has 
points which are not part of 1Z a . Consider the following 




Figure 1. Pictorial representation of Theorem Q~] The triangle denotes 
the simplex of all joint distributions over (Xi, X%, . . . , Xjv). The 
black star denotes the joint distribution representing the product of 
marginals (codebook generation). Each loop represent the set of all 
joint distributions satisfying the conditions imposed on {X}j for 
some J . The intersection of all the loops (red region) represents 
the set of joint distributions satisfying all the conditions. The blue 
stars represent the joint distributions which maximize functionals 
H(P({X}j)) (equivalently, minimize the relative entropy with the 
product of marginals as seen from Sanov's theorem) subject to 
the conditions on {X}j. Theorem [T] asserts that a separate joint 
distribution for each J can be chosen from the corresponding loop 
(blue stars) and hence all the functionals H(P({X}j)) can be set 
to their respective maxima simultaneously. 



| (xj,x 4 ) | 0,0 | 0,1 | 1,0 | i,~H 
| P(x„x 4 ) | 1/2 | | 1/4 | 1/4 | 

Table I 

Pairwise PMF of (Xi, Xa) Vi e {1, 2, 3} 



example of 4 binary random variables (Xi, X 2 , X 3 , X 4 ). 
X\,X2 and X 3 are distributed bern(^) and X4 is 
distributed 6ern(|), where bern(p) denotes a Bernoulli 
random variable with -P(O) = p and -P(l) = 1 — p. Let 
S 2 ■ ■ ■ j Se be all possible subsets of {1,2,3,4} of 
cardinality 2. Let P Sj ({X} Sj ) be such that (Xi,X 2 ,X 3 ) 
are pairwise independent and the pairwise PMF of 
(Xi,X 4 ) Mi E {1,2,3} is given in Table U Note that 
these pairwise densities are satisfied by at lease one 
joint density obtained by the following operations : 
X 3 = X\ ©X 2 and X 4 = X% • X 2 , where X\ and X2 are 
independent 6ern(|) random variables and '©' and '•' 
denote 'bit-exor' and 'bit-and' operations respectively. 

Observe that maximizing the entropy over 
(Xi,X 2 ,X 3 ) subject to their respective pairwise 
densities makes them mutually independent. However, 
there exists no joint distribution over (X\, X 2 , X 3 , X4) 
satisfying all the pairwise conditions which makes 
(Xi, X 2 , X3) mutually independent. This intuition is 
in fact sufficient to see that 1Z* D TZ a . However to be 
more rigorous, we first rewrite the achievable region 



7Z* for this example as: 

Ri + Ri > H h (\)-±H b {\) 

Ri + Rj + i? 4 > 2 + H b (~)-H*(X i ,X j ,X 4 ) 

4 1 
J^Ri > 3 + J f7 fe (-)-F*({X} li2 , 3 ,4X24) 
i=l 

Vi,j € {1,2,3} where #&(■) denotes the binary entropy 
function and {X}i j2 ,3,4 = {^1,^2,^3,^4}- 

We consider the following corner point of (1241 . A = 
(0, 0, 0, 3+iJ 6 (i)-i?*({X}i j2) 3,4)). It is sufficient for us 
to prove that A^lZ a . Note that, if R X = R 2 = R 3 = 0, 
(X\ : X 2 ,X 3 ) must be mutually independent (which in- 
turn satisfies the pairwise independence conditions). To 
prove that A ^ lZ a , we will show that there cannot 
exist any joint PMF over (Xi, X 2 , X3, X4) satisfying 
all pairwise distributions and for which (Xi,X 2 ,X 3 ) 
are mutually independent. Let us suppose that such a 
joint PMF exists. Denote the conditional PMF P(X 4 = 
0\xi,x 2 ,x 3 ) = a XlX2X3 , xi,x 2 ,x 3 € {0,1}. As 
(X±,X2,X 3 ) are assumed to be mutually indepen- 
dent, the joint distribution Px 1 ,x 2 ,x 3 ,x 4 (xi,X2, x 3 ,l) = 
1 pajrwise distribution of (X\,X 4 ) (from 

Table U) is such that P Xl ,x 4 (0, 1) = V* E {1,2,3}. 
This leads to the conclusion that a XlX2X3 = 1 if any 
one of xi,X2,x 3 is 0. We are only left with find- 
ing am. Further, we want P Xl ,x A (l,l) = j> i- e - 
T, X2 ,x 3 P x 1 ,X 2! x 3 ,X i {l,X2,x 3 ,l) = Ex 2 ,x 3 ~ Q 8 X2X3 = 
\. One substituting, we have am = 2. As a Xl>X2>X3 s 
are conditional probabilities, this leads to a contradiction 
and proves that there cannot exist a joint distribution 
with (Xi, X2, X 3 ) being mutually independent. There- 
fore 1Z* D lZ a , proving the second half of the Theorem. 

■ 

III. Application to Multi-User Gray-Wyner 
Network 

We finally apply the results in Theorem [T] to obtain 
a new achievable region for the multi-user Gray-Wyner 
network. To illustrate the applicability and to maintain 
simplicity in notation, we only consider the 3-user loss- 
less Gray-Wyner network here. However the approach 
can be extended directly to the general L— user setting 
and to incorporate distortions. Note that the formal 
definition of an achievable rate region closely resembles 
that in (SJ, with obvious generalization to the 3 user 
setting as shown in Fig. [2] We omit the details here due 
to space constraints. We further note that the rate region 



r-<-Rl3 

^ % /?3 



5 A'i 



*X 2 



*X 3 



Figure 2. 3-user Gray-Wyner network: There is a unique branch 
from the encoder to every subset of the decoders 



is in general 7 dimensional, with the following rates: 

(R\ , R 2 , ^3 , Rl2, Rl3, -C?23, Rl23J- 

Corollary 1. Let {X\,X 2 ,X 3 ) be the random variables 
with joint distribution P(X\,X<2,X 3 ) observed by the 
encoder. Let (C/123, C/12, U\ 3 , U23) be random variables 
jointly distributed with (Xi,X 2 ,X 3 ) with conditional 
distribution P(U\2 3 , U12, U\ 3 , C^23|-X"i , X 2 , X 3 ) and tak- 
ing values over arbitrary finite alphabets. Define sub- 
sets Si = {C/123, U 12 ,Ui 3 }, $2 = {^123,^12,^23}, 
53 = {C/123, C/13, C/23}. The rate region for the 3-user 
lossless Gray-Wyner network contains all the rates such 
that y(i,j,k) € {1,2,3} and i < j, i < k, 



R\23 
R\23 + Rij 



> 
> 



Rl23 + Rij + Rik — 



R\23 + Ri. 

i<j 



Ri 



> 



> 



H(U 123 )-H*(U 123 \X) 

—H*(Ui23, Uij\X) 
H(U 123 ) - H*({U} 123>ij}ik \X) 
+H(U tJ \U 123 ) + H(U lk \U 123 ) 
H{U 123 )+J2 H iUij\U 123 ) 

i<j 

-H*(U 123 ,U 12 ,U 23 ,U 13 \X) 
H(X l \{U} J:ieJ ) (25) 



where X 
by: 



{Xi,X 2 ,X 3 } and H*({U}j\X.) is given 

(26) 



max H[P({U}j 

P({U}j,{X} lt2 , 3 ) 



X 



where P({U}j\'X.) satisfies: 

P {{U} JnSj , Xj) = P {{U} JnS] , Xj) Vj (27) 

The closure of the achievable rates over all conditional 
distributions P(Ui 23 ,Ui 2 , Ui 3 ,U 23 \Xi, X 2 , X 3 ) is an 
achievable region for the 3-user lossless Gray-Wyner 
network. 



Proof: A codebook for U\ 23 consisting of 2 nRl23 
codewords is generated according to the marginal 
P( C/123). Conditioned on each codeword of C/123, inde- 
pendent codebooks are generated for Ui 2 , U\ 3 and U 23 at 
rates of i?i2 , -R13 and i?23 according to their respective 
conditional distributions P {Ui 2 \U\ 23 ), P{U\ 3 \Ux 23 ) and 
P{U 23 \U\ 23 ). If the rates satisfy (|25T ). then there always 
exists a codeword tuple, one from each codebook, de- 



noted by 



l 123> 



Ui 2 , 1*13, ^23)' sucn tnat tne following 



subsets of sequences are jointly typical according to their 
respective subset joint densities: (x", «™ 2 3, u i2i ^13)' 

(»2> u 123> u 5j>«23) and (^3^123^13^23)- The P roof 

follows rather directly from Lemmas [T] and Theorem 
[Das C/123 is part of S\,S 2 and ^3. The last constraint in 
(|25T ) denotes the minimum rate of the bin indices required 
to achieve lossless reconstruction at each sink given that 
all the codewords received at any sink are jointly typical. 



IV. Discussion 



We note that the conditions in d25l ) ensure joint typi- 
cality of source sequence Xf only with the codewords 
which reach sink i. However an alternate achievable 
region (which is subsumed in the above region) can 
be derived using results of the general L— channel MD 
problem in which extends the principles underlying 
© to the multiple descriptions framework. Due to the 
inherent structure of the MD problem, joint typicality 
of all the transmitted codewords is necessary. However 
imposing such a constraint limits the performance of 
systems that do not explicitly require such conditions. 
Note that, although we have not proved formally that 
the new region for the multi-user Gray-Wyner network 
is strictly larger than that derivable from the results in 
0, Theorem |2] suggests that for general sources, there 
exist points which are strictly outside. It is important 
to note that implications of the results we derived may 
not always lead to a strictly larger achievable region. A 
classic example of this setting is the 2 user Gray-Wyner 
network [8 1 for which the complete rate-distortion region 
can be achieved even if joint typicality of all the code- 
words is imposed. This is because, in the 2-user scenario, 
there is no inherent conflict between maximum entropy 
distributions of different subsets of random variables. 
However, in the L— user setting (as seen in Theorem 
12), such a conflict arises and maintaining joint typicality 
only within subsets plays a paramount role in deriving 
improved achievable regions. 



References 



[I] A. El-Gamal, Y.H. Kim, "Lecture notes on network information 
theory", 23-61 to 23-67, [Ettp7/arxiv.org/abs/1001.3404| , 2010. 

[2] R. Venkataramani, G. Kramer, V.K. Goyal, "Multiple descrip- 
tion coding with many channels", IEEE Trans, on Information 
Theory, vol.49, no.9, pp. 2106- 2114, Sept 2003. 

[3] A. El Gamal and T. M. Cover, "Achievable rates for multiple 
descriptions," IEEE Trans. Inf. Theory, vol. IT-28, pp. 851-857, 
Nov. 1982. 

[4] R. Puri, S. S. Pradhan, and K. Ramchandran, "n-channel 
symmetric multiple descriptions-part II: an achievable rate- 
distortion region", IEEE Trans. Information Theory, vol. 51, 
pp. 1377-1392, Apr. 2005. 

[5] K. Viswanatha, E. Akyol and K. Rose, "Combinatorial message 
sharing for a refined multiple descriptions achievable region", 
in Proc. IEEE Symp. Information Theory (ISIT), Aug. 2011. 

[6] J. Nayak, S. Ramaswamy, K. Rose, "Correlated source coding 
for fusion storage and selective retrieval", in Proc. IEEE Symp. 
Information Theory (ISIT), Sept. 2005. 

[7] K. Viswanatha, E. Akyol and K. Rose, "An achievable rate 
region for distributed source coding and dispersive information 
routing" in Proc. IEEE Symp. Information Theory (ISIT), Aug. 
2011. 

[8] R. Gray and A. Wyner, "Source coding for a simple network", 

Bell systems technical report, Dec 1974. 
[9] R. Tandon, L. Sankar, and H. V. Poor, "Multi-user privacy: The 

Gray- Wyner system and generalized common information," in 

Proc. IEEE Symp. Information Theory (ISIT), Aug. 2011. 
[10] T. Cover and J. Thomas, "Elements of Information Theory", 

Wiley publications, Second edition, 2006. 

[II] E. Perron, S. Diggavi, E. Telatar, "On the role of encoder side- 
information in source coding for multiple decoders," In Proc. 
IEEE International Symposium on Information Theory (ISIT), 
vol., no., pp.331-335, 9-14 Jul 2006. 



